Pull requests must include at least one of the required labels: internal (no release notes required), highlight, enhancement, bug, deprecation, documentation. Except for internal, pull requests must also include a description in the release notes section.

Pull requests must include at least one of the required labels: internal (no release notes required), highlight, enhancement, bug, deprecation, documentation. Except for internal, pull requests must also include a description in the release notes section.

Pull requests must include at least one of the required labels: internal (no release notes required), highlight, enhancement, bug, deprecation, documentation. Except for internal, pull requests must also include a description in the release notes section.

PR Summary

This PR refactors the test loading functionality to improve modularity and support for custom scorers. The primary changes include:

1. Extraction of helper functions:

   • A new helper function _get_test_function_from_provider is introduced to first check the scorer_store for a custom scorer based on the test ID, and if not found, fall back to retrieving the test function from the corresponding provider. This encapsulates the logic for custom scorer lookup and error handling when a test provider is missing, or when the test cannot be loaded.

   • A new helper function _configure_test_function is added to set up the test function, including adding the test ID, ensuring a proper docstring, attaching input parameters, and guaranteeing the existence of tags and tasks attributes. This simplifies the main load_test function by abstracting the configuration details.

2. Updates to the load_test function:

   • Refactoring of the logic to use the new helper functions, streamlining the process of loading and configuring a test function. With the improved workflow, the function now checks for a custom scorer before falling back to the provider-based test loader.

3. Enhancements to test IDs listing:

   • The _list_test_ids function now aggregates test IDs from multiple sources including the default test provider and custom scorers from scorer_store. It also integrates scorers provided from a built-in provider if available, ensuring a comprehensive list of all available test IDs.

Overall, these changes improve code readability and maintainability while extending functionality to support custom scorer registrations alongside traditional test providers.

Test Suggestions

• Verify that a custom scorer from the scorer_store is loaded successfully when present.
• Test that the fallback mechanism correctly loads the test function from the test provider when a custom scorer is not found.
• Ensure that a LoadTestError is raised when a non-existent test provider is referenced.
• Check that the test function is properly configured with the expected attributes (test_id, docstring, inputs, params, tags, and tasks) after calling load_test.
• Validate that _list_test_ids returns a complete and correctly sorted list of test IDs from both the provider and scorer_store.